Virtual Reality (VR) involves the use of computer technology to simulate a users physical presence in a virtual environment. Typically, VR rendering devices make use of Head Mounted Displays (HMD) to render the virtual environment to the user, although other types of VR displays and rendering techniques may be used as well, including but not limited to holography and Cave automatic virtual environments.
It is known to render VR video using such VR rendering devices, e.g., a video that is suitable for being played-out by a VR rendering device. The VR video may provide a panoramic view of a scene, with the term ‘panoramic view’ referring to, e.g., an at least 180 degree view. The VR video may even provide larger view, e.g., 360 degrees, thereby providing a more immersive experience to the user.
A VR video may be provided to a VR rendering device in a streaming process. In such a streaming process, image data of the VR video is transferred from a source to, ultimately the display of the VR device. For example, a VR video may be streamed to a VR rendering device as a single video stream. However, if the entire panoramic view is to be streamed in high quality and possibly in 3D, this may require a large amount of bandwidth, even when using modern video encoding techniques. Such bandwidth requirements may easily reach tens or hundreds of Mbps. As VR rendering devices frequently stream the video stream via a bandwidth constrained access network, e.g., a Digital Subscriber Line (DSL) or Wireless LAN (WLAN) connection or Mobile connection (e.g. UMTS or LTE), the streaming of a single video stream may place a large burden on the access network or such streaming may even not be feasible at all. For example, the play-out may be frequently interrupted due to re-buffering, instantly ending any immersion for the user. Moreover, the receiving, decoding and processing of such a large video stream may result in high computational load and/or high power consumption, which are both disadvantageous for many devices, esp. mobile devices.
It has been recognized that a large portion of the VR video may not be visible to the user at any given moment in time. A reason for this is that the Field Of View (FOV) of the display of the VR rendering device is typically significantly smaller than that of the VR video. For example, a HMD may provide a 100 degree FOV which is significantly smaller than, e.g., the 360 degrees provided by a VR video.
As such, it has been proposed to stream only parts of the VR video that are currently visible to a user of the VR rendering device. For example, the VR video may be spatially segmented into a plurality of (usually) non-overlapping video streams which each provide a different view of the scene. When the user changes viewing angle, e.g., by rotating his/her head, the VR rendering device may determine that another video stream is needed (henceforth also simply referred to as ‘new’ video stream) and switch to the new video stream by requesting the new video stream from a VR source.
Due to adapting and executing the streaming process unavoidably delays occur between a head movement and adapting the displayed image data to the new head orientation, i.e. ultimately the right photons arriving at the eye. Disadvantageously, the delay between the user physically changing viewing angle, and the new view actually being rendered by the VR rendering device, may be too large. Such delay may be called ‘motion-to-photon latency’ in general or ‘switching latency’ when different video streams are involved, and may be sizable due to an aggregate of delays. The delay between requesting a new video stream and the new video stream actually arriving at the VR rendering device is typically a large contribution. The rendering device may have to wait for a decodable frame due to the random access time in the new stream, for example an I-frame in a Group of Pictures, before decoding can start. Other, delays include buffering delays, synchronization delays, delays due to the decoding of the video streams, delays in the measurement of head rotation, etc.
Various attempts have been made to address the latency problem. For example, it is known to segment the plurality of video streams into at least partially overlapping views, or by using video tiles to send more image regions than the current viewport, thereby providing so-termed ‘guard bands’ which contain video content just outside the current view. Viewport is the part of the VR data that is actually looked at by the viewer. The size of the guard bands is typically dependent on the speed of head rotation and the latency of switching video streams. Disadvantageously, given a particular bandwidth availability, the use of guard bands reduces the video quality given a certain amount of available bandwidth, as less bandwidth is available for the video content actually visible to the user. It is also known to predict which video stream will be needed, e.g., by predicting the user's head rotation based on inertia and a current rotation speed, and request and stream the new video stream in advance. However, as in the case of guard bands, bandwidth is then also allocated for streaming non-visible video content, thereby reducing the bandwidth available for streaming currently visible video content. Also, it is not possible to anticipate on movement starting using such predictions, only to extrapolate existing movement.
US20150346832A1 describes a playback device which generates a 3D representation of the environment which is displayed to a user of the customer premise device, e.g., via a head mounted display. The playback device is said to determine which portion of the environment corresponds to the users main field of view. The device then selects that portion to be received at a high rate, e.g., full resolution with the stream being designated, from a priority perspective, as a primary stream. Content from one or more other streams providing content corresponding to other portions of the environment may be received as well, but normally at a lower data rate.
A disadvantage of the playback device of US20150346832A1 is that it may insufficiently reduce switching latency. Another disadvantage is that the playback device may reduce the bandwidth available for streaming visible video content.